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ABSTRACT 

The P.E.P. Report 1969-1973 focuses on the various 
findings and activities of the Program Evaluation Project. The study 
in this chapter was designed to conduct a statistical analysis of the 
Goal Attainment Score, and estimate variance components due to choice 
of material in the followup guide, followup interviewer bias or 
error, and the client's actual long-term deviation from expectation. 
These factors together determine the reliability of the Goal 
Attainment score as it was applied in this Program Evaluation Project 
study, and, in addition, provide somo useful indication of its 
potential reliability in other evaluative applications. 
(Author/EC) 
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GENERAL INTRODUCTION TO THH P,E.P, REPORT 1969>1973 



The P-E.P, Report 1969-1973 focuses on the various findings and activities of the Program Evaluation 
Project. It is being published in pamphlet form with one pamphlet for each chapter. 

As of January, 1974, the Program Evaluation Project, whose title was changed to the Program Evaluation 
Resource Center as of June, 1974, is funded by a three year collaborative grant with the Mental Health 
Services Division of the National Institute of Mental .4ea1th. The purpose of the grant is to emphasize the 
coordination and dissemination of information on a variety of program evaluation methodologies, especially 
Goal Attainment Scaling. 

Further information on the Goal Attainment Scaling methodology and program evaluation is available in 
other written and recorded materials from the Program Evaluation Resource Center office. At this time 
various other chapters of the P.E.P Report 1969-1973 -are available, including Chapter One, "Basic Goal At- 
tainment Scaling Procedures", Chapter Two, "Activities of the Follow-up Unit", Chapter Three, "An Intro- 
duction to Reliability and the Goal Attainment Scaling Methodology", Chapter Five, "A Construct Validity 
Overview of Goal Attainment Scaling" and Chapter Nine, "Evaluation of the Adult Outpatient Program, Hennepin 
County Mental Health Service". Additional chapters will be released this year as they are completed. 
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SYNOPSIS FOR CHAPTER FOUR 
AN EXAMINATION OF THE RELIABILITY OF THE KIRESUK-SHERMAN 
GOAL ATTAINMENT SCORE BY MEANS OF COMPONENTS OF VARIANCE 



PURPOSE : The study in this chapter was designed to conduct a statistical analysis of the Goal Attainment 
Score, and estimate variance components due to choice of material in the follow-up guide, follow-up inter- 
viewer bias or error, and the client's actual long-term deviation from expectation. These fact'^ together 
determine the reliability of the Goal Attainment score as it was applied in this Program Evalua i Pro- 
ject study, and, in addition, provide some useful indication of its potential reliability in other evalu- 
ative applications. 



MAJOR FINDINGS : Two Goal Attainment Follow-up Guides were independently completed on each of 44 clients. 
Eaich client was followed-up twice by different follow-up interviewers, and each follow-up guide scored 
on each occasion. Thus, each client yielded four Goal Attainment scores. Analyzing these data by a com- 
ponents of variance model yielded estimated score variances of 47.70 (50^) due to client long-term deviation 
from expectation, 14.53 (1555) due to short-term client changes or follow-up bias fluctuations, 16.12 (17%) 
due to choice of follow-up guide material, and 17.93 (18%) due to follcw-up interviewer errors in scoring 
or observation. 

These findings are then related to various suggested modifications in the Goal Attainment Scaling procedure. 
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I. I NTRODUCTION 

The purpose of the study on Goal Attainment 
Scaling by the Program Evaluation Project staff 
was to examine the feasibility of shifting the 
emphasis in program evaluation away from process 
factors (such as volume, load, etc.) toward 
measures of outcrmie reflecting attainment of in- 
dividualized clinical goals (alleviation of de- 
pression, vocational adjustment, etc.). This 
report presents a detailed discussion of one 
reliability study of the Goal Attainment Scaling 
methodology utilized at the Hennepin County 
Mental Health Service. 



A. Goal Attainment Scaling Methodology, G <^neral 

The Goal Attainment Scaling methodology is 
a client-specific method of goal setting and 
evaluation. The methodology allows the goal 
setter to establish unique goals and levels of 
attainiryent for individual clients while retain- 
ing the ability to make outcome comparisons. 
Its basic characteristics are: 1) establishing 
a set of specific goals with or for the client; 
2) assigning weights (w^) to each goal relative 
to its outcome significance; 3) projecting a 
follow-up date; and 4) establishing a well-de- 
fined set of attainment levels for each pro- 
jected goal. At the prespecified follow-up 
date the levels of attainment (x^) on all 
specified goals are determiiisd. These attain- 
ment levels, given values from -2 to +2, and 
the relative goal weights (any set of positive 
values), are used to generate a standardized 
Kiresuk-Shemian "Goal Attainment score", Y. 



Y = 50 + ^Q^^i^l , 

/(l-p)EW^.2 + p(EWi.)2 



where p is taken to be .3. 



B. Goal Attainment Scaling Methodology, As 
Used At Hennepin County Mental Health 
Service 

In the application of Goal Attainment 
Scaling at the Hennepin County Mental Health 
Service, follow-up guides were constructed for 
all new clients during the intake process. 
This intake process consisted of one or two 
diagnostic interviews, usually included com- 
pletion of psychological testing and, when 
necessary, a medication consultation. It was 
the intake clinician's responsibility to com- 
plete a follow-up guide with a minimum of three 
goals for each intake case, a typical follow- 
up guide constructed for use in the research 
study is shown in Figure I. 

Care was taken to insure the "follow-up- 
ability" of the goals on the follow-up guides. 
The follow-up guides were reviewed by menbers 
of the research staff for problems which might 
interfere with the scoring of the follow-up 
guides. Problems were negotiated with the 
follow-up guide constructor for clarification 
or change. Clients were then assigned to a 
treatment mode. The assignment was random, 
if ethically possible. 



FIGURE I: Sample Goal Attainment Follow-up Guide 





1 

GOAL ATTAINMENT FOLLOW-UP GUIDE 


L«v«l« of 


F«i11y C(Majn1c«tfon 


seal* ^ wt. - _3_ 
A^Utl;i9 problem 


Sc«l« ,_3_ wt^ - 1 
Education 


Stale i__ wt. - jd_ 
living Arrangements 


Outcooa Thouqht 
Lik«ly with TliAripv 


I rtfutt to itty In tiM 
rocm with iny parents «t 
all Leave rooM 1*^ 
nedlately. 


i can't <dKit that I have 
difficulties to anyone 
except iqyself. 


No desires and no y\ini to 
go back to s-chool . 


I live alone in an 
apartiient or single 
roon. 


Leas Th«n 
Expected Success 
w'lth 7h.er*py 


uni not stay In sane 
room with parents for 
more than 10 mintues. 


I can admit that I have 
sone physical problefns, 
but no emotional problems. 


I want to go back to 
school , but h;ive no 
srecific plans for doing 
so. 


I M ve with ny parents . 


Hx'.i'CCtcd level 
of Saccess 
With Therapy 


Will stay In same roon 
with parents for 11 to 
20 nintues. 


I can aiffflit only one or 
two days per week that I 
have one enotlonal prob- 
lem. 


I want to go back to 
school , and have nade 
plans (collected infor- 
naticn on school, thought 
about courses). 


I live with relatives 
other than ny parent:. 

1 


Kore Than 
fcxoectuftl success 
with Therapy 


UiU stay In saM room 
with parents for faore 
than 20 n1ngtes» but 
only If soffjeone else fs 
present. 




* wdfit to go to schco! | I live with ore or 
and hjve fr^dc plani and ;rore non-rcljtives but 
have enrolled In one Idcr't have close fricr.d- 
coursc. ships with any of their.. 


Most ravoTAhle 
-;itel'/ With Therapy 


wni stay in same roon 
with parents for nore 
than 20 nirutes, even if 
no one else is around. 


I :an adnit alrcst any ' As atiOvc, ard have en- 
day tr.it I h2ve rorc ro'lcd in more than one 
tran are errcticnal prob- 1 course. 

. i 

1 


I live v.itn nor.-relati ves| 
and have a close frienC^ | 
ship witn at least one j 
of them. 

1 
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At the specified follow-up time, "moon- 
lighting" social workers from other local 
agencies would personally interview the client 
and score the follow-up guide. These scores 
were withheld from the Mental Health Service 
staff until the conclusion of the study. 



C. The Relationship of Reliability to Validity 
for the Goal Attainment Score 

Under suitable assumptions, Sherman (1974) 
has observed that the validity of the Goal 
Attainment score can be established through th« 
content validity argument. This argument con- 
cludes that the Goal Attainment score, by its 
nature and by what it is represented to measure, 
is as valid as it is reliable. This conclusion 
emphasizes the importance of a detailed examina- 
tion of the Goal Attainment score reliabilit:. 



II. Study Objectives and Design 

A satisfactory appraisal of the reliability 
of the Goal Attainment score must address at 
least the following questions: 

a. What is the total amount of variation 

of Goal Attainment scores in the measured 
population? 

b. How much of the total variation is due 
to the particular Goal Attainment 
Follow-up Guide that happened to have 
been made for a c if lent (i.e., if a 
client had seen a different intake in- 
terviewer, an altogether different Goal 
Attainment Follow-up Guide might have 
been made)? 

c. How much of the total variation is due 
to observation or scoring errors in 
follow-up? 

d. How much of the total variation is due 
to the particular moment of the follow- 
up interview? (In our case, follow- 
up interviews were made about six monu/f^ 
after assignment to treatment; one 
would hope that choosing five or seven 
months instead, would have little effect 
on the outcome measure.) 

e. Finally, how much of the total varia- 
tion can be assigned to the client, in- 
dependent of the particular Goal Attain- 
ment Follow-up Guide, follow-up time, 
and observation error? The element 
creating this variation is what we are 
trying to measure. 

To answer these questions efficiently, an 
analysis of variance model was chosen that re- 
quired two follow-up guides on each subject, 
and two follow-ups on each fallow-up guide. 
Thus, each subject would yi .d four Goal Attain- 
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ment scores, one from each follow-up guide on 
each follow-up interview. It was judged that 
sufficient accuracy could be achieved with 40 
subjects. 

All adult outpatients of the Mental Health 
Service would have follow-up guides constructed 
for them during the intake process. The second 
follow-up guide required for the reliability 
study would be obtained from the assigned 
therapist. The therapist would tailor his follow- 
up guide to the follow-up date specified by the 
intake interviewer (usually six months to a 
year after treatment assignment) but would be 
other^vise unaware of the material on the intake 
interviewer's follow-up guide. 

To insure that each follow-up guide received 
about equal attention in the follow-up inter- 
view, and to minimir:^ the likelihood of a follow- 
up interviewer recognizing the follow-up guide 
origin from its content, the scales from the tw 
follow-up guides were randomly mixed and typed 
on a single master follow-up guide. (The scales 
were separated later for the analysis.) 

At approximately the prespecified follow- 
up date, the master guide would be scored simul- 
taneously in a follow-up interview and then 
scored again in another follow-up interview (by 
a different interviewer) about two weeks later. 



III. Results 



A. Course of the Study 

From May 1970 to October 1972, dual follow- 
up guides were completed on 84 clients. Of 
these, 44 were successfully followed-up twice. 
The reasons for the failures were: 17 clients 
were unlocatable for either the first or second 
follow'up interview; 15 clients refused to 
participate in either the first or second follow- 
up interview; and for eight clients, other 
criteria were not t. such as poor follow-up 
gu^'de construction on clients not having com- 
pleted the minimum of two therapy sessions in 
their assigned mode prior to the prescribed 
follow-up date. 

Of the 44 successfully followed-up subjects, 
29 (66%) were female, and ages ranged from 18 
to 52, with an average age of 27. These and 
other client characteristic* are similar to 
those of the rest of the Mental Health Service 
client population. (More detail can be found 
in chapter six of the P.E.P. Report, 1969-1973. ) 

Subjects were treated by Individual Therapy 
(33, or 75%); Group Therapy (6, or 14%); 
Marriage Counseling (3, or 7%); Day Care Treat- 
ment (1, or 2%); and Medication Clinic (1, or 
2%). The professions of the Mental Health 
Service staff were represented in both the in- 
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take interview and therapy functions. Most were 
social workers though psychiatrists, psychologists 
and psychiatric nurses also participated in 
approximate proportion to their numbers on the 
Mental Health Service staff. 

The length of time between the first and 
second follow-ups ranged from 5 to 67 days, with 
a mean of 25 days (see Table I). To investi- 
gate the effect of time between follow-ups on 
the size of the difference between Goal Attain- 
ment scores from the two follow-up times, all 
clients* differences in average Goal Attainment 
scores at first and second follow-ups (absolute 
values) were ranked; times between folic. .-ups 
were ranked; and a Spearman rank order correla- 
tion coefficient was computed. The value was 
rs = .12 (N = 44), far from significance. 

The Goal Attainment score on either follow- 
up guide from either follow-up had means and 
standard deviations close to the expected values 
of 50 and 10, respectively (see Table I). Table 
I also gives the means for the sample total, as 
well as means for a breakdown of the sample by 
the number of days between follow-ups. 

TABLE I: Mean Goal Attainment Scores for Both 
Follow-up Interviews and Both Follow- 
up Guides by Number of Days Between 
Follow-up Interviews 



.; j-Virr of (i.tvi i.-'-.vcn 
f i r ; I I t fi . r . - i <i i C- f 7 1 ; 




n-29 




43-67 


TOTAL 


Nurtiicr of fiubJ^ClSJ 


!; - I? 


:j ^ 17 


i; " 11 


N - 4 


ri « 44 














ItttvtVe IritorvU-wer C.A.S. 


43.33 


30.97 


47.61 


30.62 


48.62 
S.D.- 9.18 


Thrroplut C.A.S. 


«8. n 


i5.36 


47.33 


54.77 


3J.43 
S.D.- 9.84 


IitiaKe Interviewer C.A.S. 




31.87 




SO. 6? 


49.83 
S.D.-11.18 


ThcraplNt C.A.S. 


49.04 


33.68 




34.37 


53. •.7 
S.D.- 8.89 



TABLE II: Counts of Clients by First and 
Second Follow-up Interviewers 

SECOND FCH.LOW-UP IMTrRVIfW 



Inlervicwet 
Code 


A 


D 


C 


D 


E 


F 


G 


Total 
Tollow-ups 


Intake 
Interviewer 
Mtrtn G.A.S. 


A 




1 




1 




4 


1 


7 


48.83 


B 








1 


1 


1 


1 


4 


46.89 


C 


1 






1 




1 




3 


43.17 


D 




3 


3 




2 


1 




9 


50.01 


E 




4 




1 






1 


6 


47.33 


f 


1 


6 




2 






3 


12 


47.01 


G 

Total 

Fol W-u.li 


2 


1 




1 




1 




3 


55.73 


15 


3 


7 


3 


8 


6 


44 


48.62 


inUke Intv. 
Medn G.A.S. 


00.09 


46. 3B 


J9.57 


46.37 


17.87 


50. 9S 


53.63 


49.82 
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All follow-up interviews were conducted by 
master's level social workers. In no case were 
both follow-up interviews conducted by the same 
interviewer, and though a random assignment of 
follow-up interviewers was not implemented, an 
attempt was made to avoid consistent linkages 
between first and second follow-up interviewers 
(see Table II). Simple analyses of variance 
did not show statistically significant differ- 
ences in average scores by follow-up inter- 
viewers. 

B. The htodel for Analysis 

In order to use analysis of variance methods 
to identify variance components for the Goal 
Attainment score, it is necessary to specify a 
detailed statistical model: 

Let Y^ju represent the Goal Attainment score 
from the kth follow-up on the jth follow-up guide 
on the ith patient. We then define the model: 

Yljk • »j + ai + + Yk + Cop)ij +Cot)u + ^»Y)jk * Mjk« 

where i goes from 1 to I (1=44), j goes from 1 to 
o (J=2), and k goes from 1 to K (K=2)« and we 
assume 

u is a true mean effect, 

a.j are random effects representing the ith 
client's true long-term average deviation from 
u, and the are NID (Normally and Independently 
Distributed) {O^a^ ^). 

6j are fixed effects representing the differ- 
ent sources of follow-up guides (the first one 
created by the intake worker, or the second one 
created by the therapist), and jBj = 0. 

Yk are fixed effects representing the effect 
of the follow-up order, that is, a combination 
of experience effect and true average client 
change across time from first to second follow- 
up, and ?Yl = 0- 
k ^ 

(a6)ij are random effects due to the jth 
guide on the ith client, and represents a devia- 
tion from a conceptual average score of an in- 
finite number of independently created follow- 
up guides on the same individual, and the 
(a6)ij are NID (D,. a^^^). 

(aY)ik random effects due to either 
true fluctuations in the state of the client 
from time to time, or fluctuations in the 
"optimism" of the follow-up interviewers from 
time to time, and the (oiY)ik NID (0, )- 

(6Y)ik are fixed effects due to the inter- 
action of^TOllow-up guide source and follow-up 
time. That is, the "learning effect", or true 
average client change across time may be differ- 
ent for follow-up guides from different sources; 
and ? (6Y)j|^ = { (BY)jk = 0- 

^ijk are residual random errors of obser-j 
vation or scoring, and the e.jj|^ are NID (D, ). 



The task is now to analyze the observed 
scores in terms of the above parameters, esti- 
mating the size and testing the significance of 
the estimated variance components. 

Though the analysis of variance which follows 
at first appears to be based on a three-factor 
factorial design with one random and two fixed 
effects (and in fact the sum of squares is broken 
down in that fashion), the expected mean squares 
do not conform to that model, Becausf? of the 
assumption that the (aB) and (ay) "interactions" 
were random variables, the design has charac- 
teristics of a "nested" or hierarchical design. 

The usual F-ratio tests demonstrate statis- 
tical significance at the .01 level for the 
effects of "Individuals", "Source of Guide", 
"Individual x Source" interaction, and "Individ- 
ual X Follow-up Order" interaction. 

TABLE III: 

Analysis of Variance 

44 Subjects, Each With Four Goal Attainment Scores 
Generated According to the Reliability Study Model 



lUKCf t>r VARIATIO'* 


df 


MS 


E(MS) 


llviduols 


1-1*43 


270.04- 
























tree of Guide 




476.03*' 










1 low-up Order 


K-1-1 








2o ' 


k 


Jj,', X Source 


(I-n(J-l).43 


50.17** 


°/ 


+ 






liv. X r.U, Order 


{U1)(K-1)>4.T 


46.99" 


a * 












c 








ircc X F.U, Ord(»r 


(.J.1)(K-1)«1 


9.17 


a ^ 




4.1^^f 








c 






;1dual F.U. Crror 


(M)(J-l)():.1)--:3 


17.93 










t^Mitica It dt the n> 


01 le^c; 













C. Variance Component Estimates 

Using the analysis of variance table, the 
variance components together with 90 percent 
confidence limits on the estimates may now be 
computed. (Scheffe', 1959) 

a^^, the residual error variance due to 
errors of observation or scoring in follow- 
up, is estimated jy s^^ = 17.93, with 90 per- 
cent confidence interval 13.00 to 26.58. That 
is, we might expect a random error with a 
standard deviation of about four points in the 
Goal Attainment score due to the follow-up in- 
terviewer's errors of observation or scoring. 

c^aB^» error variance due to the con- 
struction of the Goal Attainment Follow-up 
Guide and the material chosen for inclusion is 
estimated by 



with 



= (50.17 - 17.93)/2 = 16.12, 
90 percep*" confidence interval 8.10 to 
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28.65. That is, we ,^^^g|,^ expect a random 
error with a standarci deviation of about four 
points (the square root of 16.12) in the Goal 
Attainment score due to the material chosen for 
the follow-up gui^^e. j^^^ the error component 
unique to the Goal Attainment Scaling procedure. 
A standardized 'fixeci" test ^^"'^ ^ave no such 
component, but such '^f^xed" ^^ts could be less 
relevant to a particular client's problems. 

a ^ the varianee component dre to fluctua- 
tions"Xvir ti^^e in either th« true state of the 
client, or the general optimism of the follow- 
up interviewers, i!* estimated 

s„/ - (46-99 ^ 17.93)/2 14.53. 

with 90 percent conf^^j ^e interval 6.86 to 26.25. 
To the extent that o 2 due to the true state 
of the client at th^ follow-UP time, we may not 
wish to consider it ^^^^rof • While a measure 
which Mould give the long-tefin average status of 
a client rather than exact condition at a 
particular moment mlg|,^ ^ preferred, such a 
measure cannot be app^^^^^gd without repeated ob- 
servations across tit^^ should, therefore, not 
stand against a one^ti^ measure if it only measures 
the status of ^ client at the time of the measure- 
ment. But this varl^^^g component may also be 
due to variations in the level of optimism of the 
follow-up intcrviewer^ jhat Is, how generous is 
the follow-up Intervig^gy, in his Interpretation 
of the client behavl^^,^ jp this case a ^ would 
be an error variance. 

a ^, the variance component due to differences 
among^clients 1n thei^, true long-term average de- 
viation from expectation is estimated by: 

s„^ = (270.04 - 50.17 ^ 45^99 + 17.93)/4 = 47.70, 

with 90 percent confi^g^ce H^^Hs 31.16 to 71.21. 
That is, if aH "^east^y^^^nt errors could be ex- 
cluded, we would be ig^^ ^^ith a Goal Attainment 
score standard deviation of about seven, instead 
of the 10 which is observed. 



In its intended application, the Goal Attain- 
ment score is computed from a single follow-up 
on a single Goal Att^i^j^gpt Fol low-up Guide. 
Thus, in the model f^^, ^|^g score, Yijj^, the j and 
k are always 1» Components that vary only with 
j or k are nOW const^^^ across all observations 
and absorbed Into the "true mean effect", p. The 
model then becomes: 

Yi = 1^ + «i ^ (aY)i ^ + ci, 

where the components represent the same effects 
as before, but now varying only across. 

The variance of then constructed as 

follows: 
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which may be estimated by: 



a 47.70 + 14.53 + 16.12 + 17.93 
= 96.28 

for which a 90 percent confidence interval 
may be computed to be 79.14 to 113.41. 

The components of variance can be related 
to the total variance of a Goal Attainment 
score (see Figure II), and we may respond to 
the questions posed in Section I, item C, viz., 

a. What is the total amount of variation 
of the Goal Attainment scores in the 
measured population? 

Answer: The variance of the score is esti- 
mated at 96.28, or a standard deviation of 
9.81. 

b. How much of this total variation is due 
to the pp./cicular Goal Attainment 
Follow-up Guide that happened to have 
been made for each client? 



FIGURE II 

BREAROOWH Of VARIANCE COHPOHlifTS Or TtIC 
COAL ATTAIfflENT SCOAi 
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Answer: The variance component due to the 
choice of guide material is estimated at 
16.12, or 17 percent of the total score 
vari ance. 

c. How much of total variation is due to 
errors of observation or scoring? 

Answer: The variance component due to fol- 
low-up error is estimated at 17.93, or 18 
percent of total score variance. 

d. How much of the total variation is due 
to the parti cMlar moment of follow-up? 

Answer: Here the experimental design could 
not separate short term client fluctuations 
from follow-up interviewer bias. These two 
components together contribute an estimated 
variance component of 14.53, or 15 percent 
of the total score variance. 

e. How much of the total variation can be 
assigned to the client, independent of the 
particular Goal Attainment Follow-up Guide, 
follow-up time, and observation error? 

Answer: The variance component assignable 
to differences among clients in their long- 
term deviation from expectation is esti- 
mated by 47.70, or 50 percent of the total 
score variance. 

The above information can be expressed in 
terms of various reliability coefficients, viz. 

How well does the Goal Attainment score 
reflect the long-term status of the client? 



We estimate: 



Sa2 _ 47.70 _ 50 
§F" 96728 - 



Or, how well does the Goal Attainment score re- 
flect the actual status of the client at the time 
of follow-up? Here again is the problem of ques- 
tion four, above. How much of s^y^ can we assign 
to the client status (which we wish to measure) 
and how much to extraneous interviewer bias? De- 
pending upon this division, we estimate the re- 
liability of the Goal Attainment score to be: 



Sa^ 



.50 



< .65 



Similarly, we can bracket the reliability of 
follow-jp scoring: 



ay 



sy 

And, finally, the reliability of follow-up guide 
construction when the constructors compared are in 
take interviewers and therapists is estimated to 
be: 



S^+c 2+c2 



.83 



It should be emphasized here that it is rj or r2 
that reflect the reliability of the Goal Attain- 
ment score in its application. The coefficients 
ro and might be considered "special interest" 
statistics. 



IV. Conclusions and Summary 

It is now clear that the Goal Attainment 
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score measured at least the degree to which 
a client's outcome status (on plausibly mental 
health related characteristics) conformed to 
the expectations of inental health professionals, 
The most complete picture of the score relia- 
bility is obtained by examining the variance 
co.iiponent estimates presented in the previous 
section. From these, two "reliability coef- 
ficients" were computed as candidates **o rep- 
resent the Goal Attainment score reliability, 
ri (= .50), and ro (between .50 and .65). It 
Simplifies the statement of this result to use 
an average figure of r - .57 to represent the 
reliability of the Goal Attainment Scaling 
application used in the Program Evaluation 
Project study. Clearly, more refined analy- 
sis of our data would not greatly change this 
estimate. 

Is Goal Attainment Scaling ready for practi- 
cal evaluative applications? The most critical 
point in the ^:rocess is surely follow-up guide 
construction. Without thoughtfully and skill- 
fully constructed follow-up guides, both follow- 
up guide construction and follow-up determina- 
tion errors may become too large. Even with 
considerable care (in both follow-up guide con- 
struction and follow-up) the reported reliability 
of .57 is only moderately high, though it does 
take into account all the errors encountered in 
the application. That is, both follow-up deter- 
mination errors (which includes both test-re- 
test and inter- rater differences) are accounted 
for in the reported r of .57. (Some reported 
reliability coefficients are either "alternate 
form" or "test-retest" reliability, but not both, 
and therefore may not represent the practical 
reliability of a score.) Given the severity of 
our test and the unique advantage of the Goal 
Attainment Scaling technique (i.e., completely 
individualized goals), the authors consider the 
Goal Attainment score acceptably reliable in 
the Program Evaluation Project application. 

However, the Program Evaluation Project 
application is basically research-oriented. 
Most evaluators face significantly d-fferent 
circumstances, programs, and overall objectives 
for the evaluation process. There may not be 
sufficient staff to permit independent follow- 
up guide construction and follow-up interviews, 
or it may be desired that the client set his 
own goals. Improvement of outcome rather than 
the evaluation of therapy may be the imnediate 
objective and, of course, a high cost evaluation 
program may be difficult to justify. There have 
been se»/eral attempts to modify the Goal Attain- 
ment Scaling procedure to make it more compati- 
ble with one or more such specifications. Though 
work is still in progress, it i£ useful to 
briefly consider, in light of this study, the 
reliability implication of some of the suggested 
procedure modificatioi^s. 



A. Clients Making Their Own Follow-up Guides 
If all clients were to make their own 
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follow-up guides, it could save staff time, 
remove therapist bias from the follow-up guide 
content, greatly imp^'ove follow-up guide con- 
struction reliability, and could also reduce 
errors of determination in the follow-up (the 
client should know what he meant when he speci- 
fied the scales). A step-by-step manual for the 
client to use in doing this has been developed 
(Garwick, 1973). The chief disadvantage of this 
modification is that the client may lack the skill 
or insight to determine realistic goals and attain 
mtnt levels. 



B. Negotiating the Follow-up Guide With th Client 

If the therapist were to negotiate the Goal 
Attainment Follow-up Guide with the client, we 
might hope to obtain many of the benefits of the 
client making the follow-up guirir himself (as 
suggested above) while eliminating through the 
negotiations many of the inappropriate or unreal- 
istic goals or attainment levels. This has been 
suggested by Sherman (1972) and applied by 
Lombillo, et. al . (1973). A related benefit of 
this modification is that good concrete cormuni- 
cation between therapist and client witn respect 
to therapy goals is necessarily established in 
the beginning. The chief disadvantage is that 
therapists may be suspected of developing a self- 
serving approach to the negotiation. 



C. Multiple Follow-ups 

Multiple follow-ups on Goal Attainment Follow- 
up Guides has been suggested as a way of follow- 
ing either the course of therapy or the durability 
of therapy re*"-ults. Multiple follow-ups would 
also permit the reduction of follow-up determina- 
tion error, and the smoothing of short-term client 
status fluctuations. Its chief difficulty is cost, 
along with the fact that cli'^nts may tire of 
cooperating, or be unlocitable. 



D. Therapists Conducting Their Own Follow-ups 

If the therapist were to conduct the follow- 
up, he would have the advantage of his clinical 
experience with the client to assist in the in- 
terpretation of the client's behavior, and fol- 
low-up determination error should be reduced. 
Feedback would be imnediate. He could use his 
acquired rapport to conduct inexpensive follow-up 
interviews by phone, making multiple follow-ups 
more practical. This modification suffers the 
possibility of therapist bias. 



E. Semi -Standardized Scales 

It could simplify the constructio, of the 
Goal Attainment Follow-up Guide and provide an 
easier starting point for categorizing cients 
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by follow-up guide content, if goals were 
selected from some finite list, perhaps each 
with a well -con St rue ted set of graded attain- 
ment levels to choose from. This might also re 
duce follow-up guide construction variance, and 
follow-up determination error as well. Its 
major disadvantage is that follow-up guides may 
be less relevant to the client's specific prob- 
lems. 



F. The Goal Attainme.tt Process as a Part of 
Therapy 

It has been suggested that the goal setting 
process is itself a useful part of therapy. In 
this model, reliability may be of little concern. 

Many of the modifications in the Goal Attain- 
ment Scaling procedure mentioned above are being 
attempted. While the results are not yet in, it 
does appear that Goal Attainment Scaling Is moving 
successfully from research to practical evaluative 
applications. 
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